{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "02a47dfb",
   "metadata": {},
   "source": [
    "# 习题\n",
    "## 习题8.1\n",
    "![image.png](./images/exercise1.png)\n",
    "### 采用AdaBoostClassifier分类器实现\n",
    "AdaBoostClassifier 是 scikit-learn 库中实现 AdaBoost（自适应提升）算法的类，它是一种集成学习算法，通过结合多个弱分类器来提高整体的分类性能。AdaBoost 算法会根据前一个分类器的性能来调整下一个分类器的训练数据权重，使得新的分类器能够重点关注之前分类器表现不佳的样本。\n",
    "\n",
    "下面是 AdaBoostClassifier 的一些主要参数：\n",
    "\n",
    "- base_estimator：基础估计器，即弱分类器。如果不指定，默认为 DecisionTreeClassifier 配置为最大深度为 1（即决策树桩）。\n",
    "\n",
    "- n_estimators：弱分类器的最大数量，即迭代次数。默认为 50。\n",
    "\n",
    "- learning_rate：每个分类器的权重缩减系数。如果设置得较低，需要更多的估计器。默认为 1。\n",
    "\n",
    "- algorithm：指定用于训练弱分类器的算法。可以是 'SAMME' 或 'SAMME.R'，后者通常更快且更有效。\n",
    "\n",
    "- random_state：控制随机数生成器的种子。\n",
    "\n",
    "使用 AdaBoostClassifier 的基本步骤包括：\n",
    "\n",
    "1. 导入 AdaBoostClassifier 类。\n",
    "\n",
    "2. 创建 AdaBoostClassifier 对象，并根据需要设置参数。\n",
    "\n",
    "3. 使用训练数据调用 fit 方法训练模型。\n",
    "\n",
    "4. 使用训练好的模型进行预测，调用 predict 方法。\n",
    "\n",
    "AdaBoostClassifier 中的 algorithm 参数可以设置为 'SAMME' 或 'SAMME.R'，这两种算法都是用于训练弱分类器的 AdaBoost 变体。它们的主要区别在于它们处理多类别分类问题的方式。\n",
    "\n",
    "'SAMME'：\n",
    "SAMME（Stagewise Additive Modeling using a Multi-class Exponential loss function）是 AdaBoost 的多类别扩展，它使用多类指数损失函数进行阶段性加法建模。在 SAMME 算法中，每个弱分类器对每个类别进行投票，然后根据这些投票的加权总和来更新样本权重。SAMME 算法适用于任何可以处理多类别问题的弱分类器。\n",
    "\n",
    "'SAMME.R'：\n",
    "SAMME.R（Real）是 SAMME 的一种改进版本，它使用了真实值的预测而不是类别标签。在 SAMME.R 中，每个弱分类器输出的是一个概率估计，而不是硬性分类结果。这些概率估计被用来更新样本权重，从而使得算法通常能够更快地收敛，并且在很多情况下能够达到更好的性能。SAMME.R 通常需要弱分类器能够输出类别概率。\n",
    "\n",
    "在实际应用中，'SAMME.R' 通常是首选，因为它通常比 'SAMME' 更快且更有效。然而，如果使用的弱分类器不能输出类别概率，那么可能需要使用 'SAMME'。在 scikit-learn 的 AdaBoostClassifier 中，默认算法是 'SAMME.R'。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "cc71d59c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "原始输出: [-1 -1 -1 -1 -1 -1  1  1 -1 -1]\n",
      "预测输出: [-1 -1 -1 -1 -1 -1  1  1 -1 -1]\n",
      "预测准确率：100.00%\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "import numpy as np\n",
    "\n",
    "# 加载训练数据\n",
    "X = np.array([[0, 1, 3],\n",
    "              [0, 3, 1],\n",
    "              [1, 2, 2],\n",
    "              [1, 1, 3],\n",
    "              [1, 2, 3],\n",
    "              [0, 1, 2],\n",
    "              [1, 1, 2],\n",
    "              [1, 1, 1],\n",
    "              [1, 3, 1],\n",
    "              [0, 2, 1]\n",
    "              ])\n",
    "y = np.array([-1, -1, -1, -1, -1, -1, 1, 1, -1, -1])\n",
    "\n",
    "# 使用sklearn的AdaBoostClassifier分类器\n",
    "clf = AdaBoostClassifier(algorithm='SAMME')\n",
    "# 进行分类器训练\n",
    "clf.fit(X, y)\n",
    "# 对数据进行预测\n",
    "y_predict = clf.predict(X)\n",
    "# 得到分类器的预测准确率\n",
    "score = clf.score(X, y)\n",
    "print(\"原始输出:\", y)\n",
    "print(\"预测输出:\", y_predict)\n",
    "print(\"预测准确率：{:.2%}\".format(score))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c7753a8",
   "metadata": {},
   "source": [
    "### 自编程实现AdaBoost算法\n",
    "\n",
    "代码思路：\n",
    "\n",
    "1. 写出fit函数，即分类器训练函数；\n",
    "2. 根据书中第158页例8.1，编写build_stump函数，用于得到分类误差最低的基本分类器；\n",
    "3. 根据算法第2步(a)~(c)，编写代码；\n",
    "4. 根据算法第2步(d)，编写updata_w函数，用于更新训练数据集的权值分布；\n",
    "5. 编写predict函数，用于预测数据；\n",
    "6. 【附加】编写score函数，用于计算分类器的预测准确率；\n",
    "7. 【附加】编写print_G函数，用于打印最终分类器。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d6624f52",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "class MyAdaBoost:\n",
    "    def __init__(self, tol=0.05, max_iter=10):\n",
    "        # 特征\n",
    "        self.X = None\n",
    "        # 标签\n",
    "        self.y = None\n",
    "        # 分类误差小于精度时，分类器训练中止\n",
    "        self.tol = tol\n",
    "        # 最大迭代次数\n",
    "        self.max_iter = max_iter\n",
    "        # 权值分布\n",
    "        self.w = None\n",
    "        # 弱分类器集合\n",
    "        self.G = []\n",
    "\n",
    "    def build_stump(self):\n",
    "        \"\"\"\n",
    "        以带权重的分类误差最小为目标，选择最佳分类阈值，得到最佳的决策树桩\n",
    "        best_stump['dim'] 合适特征的所在维度\n",
    "        best_stump['thresh']  合适特征的阈值\n",
    "        best_stump['ineq']  树桩分类的标识lt,rt\n",
    "        \"\"\"\n",
    "        m, n = np.shape(self.X)\n",
    "        # 分类误差\n",
    "        min_error = np.inf\n",
    "        # 小于分类阈值的样本所属的标签类别\n",
    "        sign = None\n",
    "        # 最优决策树桩\n",
    "        best_stump = {}\n",
    "        for i in range(n):\n",
    "            # 求每一种特征的最小值和最大值\n",
    "            range_min = self.X[:, i].min()\n",
    "            range_max = self.X[:, i].max()\n",
    "            step_size = (range_max - range_min) / n\n",
    "            for j in range(-1, int(n) + 1):\n",
    "                # 根据n的值，构造切分点\n",
    "                thresh_val = range_min + j * step_size\n",
    "                # 计算左子树和右子树的误差\n",
    "                for inequal in ['lt', 'rt']:\n",
    "                    # (a)得到基本分类器\n",
    "                    predict_values = self.base_estimator(self.X, i, thresh_val, inequal)\n",
    "                    # (b)计算在训练集上的分类误差率\n",
    "                    err_arr = np.array(np.ones(m))\n",
    "                    err_arr[predict_values.T == self.y.T] = 0\n",
    "                    weighted_error = np.dot(self.w, err_arr)\n",
    "                    if weighted_error < min_error:\n",
    "                        min_error = weighted_error\n",
    "                        sign = predict_values\n",
    "                        best_stump['dim'] = i\n",
    "                        best_stump['thresh'] = thresh_val\n",
    "                        best_stump['ineq'] = inequal\n",
    "        return best_stump, sign, min_error\n",
    "\n",
    "    def updata_w(self, alpha, predict):\n",
    "        \"\"\"\n",
    "        更新样本权重w\n",
    "        :param alpha: alpha\n",
    "        :param predict: yi\n",
    "        :return:\n",
    "        \"\"\"\n",
    "        # (d)根据迭代公式，更新权值分布\n",
    "        P = self.w * np.exp(-alpha * self.y * predict)\n",
    "        self.w = P / P.sum()\n",
    "\n",
    "    @staticmethod\n",
    "    def base_estimator(X, dimen, thresh_val, thresh_ineq):\n",
    "        \"\"\"\n",
    "        计算单个弱分类器（决策树桩）预测输出\n",
    "        :param X: 特征\n",
    "        :param dimen: 特征的位置（即第几个特征）\n",
    "        :param thresh_val: 切分点\n",
    "        :param thresh_ineq: 标记结点的位置，可取左子树(lt)，右子树(rt)\n",
    "        :return: 返回预测结果矩阵\n",
    "        \"\"\"\n",
    "        # 预测结果矩阵\n",
    "        ret_array = np.ones(np.shape(X)[0])\n",
    "        # 左叶子 ，整个矩阵的样本进行比较赋值\n",
    "        if thresh_ineq == 'lt':\n",
    "            ret_array[X[:, dimen] >= thresh_val] = -1.0\n",
    "        else:\n",
    "            ret_array[X[:, dimen] < thresh_val] = -1.0\n",
    "        return ret_array\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        \"\"\"\n",
    "        对分类器进行训练\n",
    "        \"\"\"\n",
    "        self.X = X\n",
    "        self.y = y\n",
    "        # （1）初始化训练数据的权值分布\n",
    "        self.w = np.full((X.shape[0]), 1 / X.shape[0])\n",
    "        G = 0\n",
    "        # （2）对m=1,2,...,M进行遍历\n",
    "        for i in range(self.max_iter):\n",
    "            # (b)得到Gm(x)的分类误差error，获取当前迭代最佳分类阈值sign\n",
    "            best_stump, sign, error = self.build_stump()\n",
    "            # (c)计算弱分类器Gm(x)的系数\n",
    "            alpha = 1 / 2 * np.log((1 - error) / error)\n",
    "            # 弱分类器Gm(x)权重\n",
    "            best_stump['alpha'] = alpha\n",
    "            # 保存弱分类器Gm(x)，得到分类器集合G\n",
    "            self.G.append(best_stump)\n",
    "            # 计算当前总分类器（之前所有弱分类器加权和）误差率\n",
    "            G += alpha * sign\n",
    "            y_predict = np.sign(G)\n",
    "            # 使用MAE计算误差\n",
    "            error_rate = np.sum(np.abs(y_predict - self.y)) / self.y.shape[0]\n",
    "            if error_rate < self.tol:\n",
    "                # 满足中止条件，则跳出循环\n",
    "                print(\"迭代次数：{}次\".format(i + 1))\n",
    "                break\n",
    "            else:\n",
    "                # (d)更新训练数据集的权值分布\n",
    "                self.updata_w(alpha, np.sign(alpha * sign))\n",
    "\n",
    "    def predict(self, X):\n",
    "        \"\"\"对新数据进行预测\"\"\"\n",
    "        m = np.shape(X)[0]\n",
    "        G = np.zeros(m)\n",
    "        for i in range(len(self.G)):\n",
    "            stump = self.G[i]\n",
    "            # 遍历每一个弱分类器，进行加权\n",
    "            _G = self.base_estimator(X, stump['dim'], stump['thresh'], stump['ineq'])\n",
    "            alpha = stump['alpha']\n",
    "            # (3)构建基本分类器的线性组合\n",
    "            G += alpha * _G\n",
    "        # 计算最终分类器的预测结果\n",
    "        y_predict = np.sign(G)\n",
    "        return y_predict.astype(int)\n",
    "\n",
    "    def score(self, X, y):\n",
    "        \"\"\"计算分类器的预测准确率\"\"\"\n",
    "        y_predict = self.predict(X)\n",
    "        # 使用MAE计算误差\n",
    "        error_rate = np.sum(np.abs(y_predict - y)) / y.shape[0]\n",
    "        return 1 - error_rate\n",
    "\n",
    "    def print_G(self):\n",
    "        i = 1\n",
    "        s = \"G(x) = sign[f(x)] = sign[\"\n",
    "        for stump in self.G:\n",
    "            if i != 1:\n",
    "                s += \" + \"\n",
    "            s += \"{}·G{}(x)\".format(round(stump['alpha'], 4), i)\n",
    "            i += 1\n",
    "        s += \"]\"\n",
    "        return s\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "fe90c5ac",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "迭代次数：6次\n",
      "原始输出: [-1 -1 -1 -1 -1 -1  1  1 -1 -1]\n",
      "预测输出: [-1 -1 -1 -1 -1 -1  1  1 -1 -1]\n",
      "预测正确率：100.00%\n",
      "最终分类器G(x)为: G(x) = sign[f(x)] = sign[0.6931·G1(x) + 0.7332·G2(x) + 0.4993·G3(x) + 0.5816·G4(x) + 0.5319·G5(x) + 0.5455·G6(x)]\n"
     ]
    }
   ],
   "source": [
    "# 加载训练数据\n",
    "X = np.array([[0, 1, 3],\n",
    "              [0, 3, 1],\n",
    "              [1, 2, 2],\n",
    "              [1, 1, 3],\n",
    "              [1, 2, 3],\n",
    "              [0, 1, 2],\n",
    "              [1, 1, 2],\n",
    "              [1, 1, 1],\n",
    "              [1, 3, 1],\n",
    "              [0, 2, 1]\n",
    "              ])\n",
    "y = np.array([-1, -1, -1, -1, -1, -1, 1, 1, -1, -1])\n",
    "\n",
    "clf = MyAdaBoost()\n",
    "clf.fit(X, y)\n",
    "y_predict = clf.predict(X)\n",
    "score = clf.score(X, y)\n",
    "print(\"原始输出:\", y)\n",
    "print(\"预测输出:\", y_predict)\n",
    "print(\"预测正确率：{:.2%}\".format(score))\n",
    "print(\"最终分类器G(x)为:\", clf.print_G())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47047ac4",
   "metadata": {},
   "source": [
    "# 自编程实现AdaBoost算法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1e576646",
   "metadata": {},
   "outputs": [],
   "source": [
    "class AdaBoost:\n",
    "    def __init__(self, n_estimators=50, learning_rate=1.0):\n",
    "        self.clf_num = n_estimators\n",
    "        self.learning_rate = learning_rate\n",
    "\n",
    "    def init_args(self, datasets, labels):\n",
    "\n",
    "        self.X = datasets\n",
    "        self.Y = labels\n",
    "        self.M, self.N = datasets.shape\n",
    "\n",
    "        # 弱分类器数目和集合\n",
    "        self.clf_sets = []\n",
    "\n",
    "        # 初始化weights\n",
    "        self.weights = [1.0 / self.M] * self.M\n",
    "\n",
    "        # G(x)系数 alpha\n",
    "        self.alpha = []\n",
    "\n",
    "    def _G(self, features, labels, weights):\n",
    "        m = len(features)\n",
    "        error = 100000.0  # 无穷大\n",
    "        best_v = 0.0\n",
    "        # 单维features\n",
    "        features_min = min(features)\n",
    "        features_max = max(features)\n",
    "        n_step = (features_max - features_min +\n",
    "                  self.learning_rate) // self.learning_rate\n",
    "        # print('n_step:{}'.format(n_step))\n",
    "        direct, compare_array = None, None\n",
    "        for i in range(1, int(n_step)):\n",
    "            v = features_min + self.learning_rate * i\n",
    "\n",
    "            if v not in features:\n",
    "                # 误分类计算\n",
    "                compare_array_positive = np.array(\n",
    "                    [1 if features[k] > v else -1 for k in range(m)])\n",
    "                weight_error_positive = sum([\n",
    "                    weights[k] for k in range(m)\n",
    "                    if compare_array_positive[k] != labels[k]\n",
    "                ])\n",
    "\n",
    "                compare_array_nagetive = np.array(\n",
    "                    [-1 if features[k] > v else 1 for k in range(m)])\n",
    "                weight_error_nagetive = sum([\n",
    "                    weights[k] for k in range(m)\n",
    "                    if compare_array_nagetive[k] != labels[k]\n",
    "                ])\n",
    "\n",
    "                if weight_error_positive < weight_error_nagetive:\n",
    "                    weight_error = weight_error_positive\n",
    "                    _compare_array = compare_array_positive\n",
    "                    direct = 'positive'\n",
    "                else:\n",
    "                    weight_error = weight_error_nagetive\n",
    "                    _compare_array = compare_array_nagetive\n",
    "                    direct = 'nagetive'\n",
    "\n",
    "                # print('v:{} error:{}'.format(v, weight_error))\n",
    "                if weight_error < error:\n",
    "                    error = weight_error\n",
    "                    compare_array = _compare_array\n",
    "                    best_v = v\n",
    "        return best_v, direct, error, compare_array\n",
    "\n",
    "    # 计算alpha\n",
    "    def _alpha(self, error):\n",
    "        return 0.5 * np.log((1 - error) / error)\n",
    "\n",
    "    # 规范化因子\n",
    "    def _Z(self, weights, a, clf):\n",
    "        return sum([\n",
    "            weights[i] * np.exp(-1 * a * self.Y[i] * clf[i])\n",
    "            for i in range(self.M)\n",
    "        ])\n",
    "\n",
    "    # 权值更新\n",
    "    def _w(self, a, clf, Z):\n",
    "        for i in range(self.M):\n",
    "            self.weights[i] = self.weights[i] * np.exp(\n",
    "                -1 * a * self.Y[i] * clf[i]) / Z\n",
    "\n",
    "    # G(x)的线性组合\n",
    "    def _f(self, alpha, clf_sets):\n",
    "        pass\n",
    "\n",
    "    def G(self, x, v, direct):\n",
    "        if direct == 'positive':\n",
    "            return 1 if x > v else -1\n",
    "        else:\n",
    "            return -1 if x > v else 1\n",
    "\n",
    "    def fit(self, X, y):\n",
    "        self.init_args(X, y)\n",
    "\n",
    "        for epoch in range(self.clf_num):\n",
    "            best_clf_error, best_v, clf_result = 100000, None, None\n",
    "            # 根据特征维度, 选择误差最小的\n",
    "            for j in range(self.N):\n",
    "                features = self.X[:, j]\n",
    "                # 分类阈值，分类误差，分类结果\n",
    "                v, direct, error, compare_array = self._G(\n",
    "                    features, self.Y, self.weights)\n",
    "\n",
    "                if error < best_clf_error:\n",
    "                    best_clf_error = error\n",
    "                    best_v = v\n",
    "                    final_direct = direct\n",
    "                    clf_result = compare_array\n",
    "                    axis = j\n",
    "\n",
    "                # print('epoch:{}/{} feature:{} error:{} v:{}'.format(epoch, self.clf_num, j, error, best_v))\n",
    "                if best_clf_error == 0:\n",
    "                    break\n",
    "\n",
    "            # 计算G(x)系数a\n",
    "            a = self._alpha(best_clf_error)\n",
    "            self.alpha.append(a)\n",
    "            # 记录分类器\n",
    "            self.clf_sets.append((axis, best_v, final_direct))\n",
    "            # 规范化因子\n",
    "            Z = self._Z(self.weights, a, clf_result)\n",
    "            # 权值更新\n",
    "            self._w(a, clf_result, Z)\n",
    "\n",
    "\n",
    "#             print('classifier:{}/{} error:{:.3f} v:{} direct:{} a:{:.5f}'.format(epoch+1, self.clf_num, error, best_v, final_direct, a))\n",
    "#             print('weight:{}'.format(self.weights))\n",
    "#             print('\\n')\n",
    "\n",
    "    def predict(self, feature):\n",
    "        result = 0.0\n",
    "        for i in range(len(self.clf_sets)):\n",
    "            axis, clf_v, direct = self.clf_sets[i]\n",
    "            f_input = feature[axis]\n",
    "            result += self.alpha[i] * self.G(f_input, clf_v, direct)\n",
    "        # sign\n",
    "        return 1 if result > 0 else -1\n",
    "\n",
    "    def score(self, X_test, y_test):\n",
    "        right_count = 0\n",
    "        for i in range(len(X_test)):\n",
    "            feature = X_test[i]\n",
    "            if self.predict(feature) == y_test[i]:\n",
    "                right_count += 1\n",
    "\n",
    "        return right_count / len(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "b2fed8d5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x31afca090>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAGdCAYAAAAvwBgXAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8fJSN1AAAACXBIWXMAAA9hAAAPYQGoP6dpAAAxxElEQVR4nO3de5BU9Z338U/33AdmmntmBoEeFCvOUlnFiGHKURajxRocyWaLhYgpEkMAyUpWKTdsxWUlhaikFC1ZNmK5USnzaJ6U9Qji4KaUSIEsRiAlDqtxmeE6AxRMd6M4F2bO80c7M8yle/pyus+vT79fVVOxm3Omv+d3Tugv5/L7eCzLsgQAAOAAr9MFAACA7EUjAgAAHEMjAgAAHEMjAgAAHEMjAgAAHEMjAgAAHEMjAgAAHEMjAgAAHEMjAgAAHJNQI/Lee+/J4/Fo586dQy5bVlYmj8fT8/PNb34zkY8EAAAulBvvCh0dHbrvvvtiXj4QCOjw4cMqKysLf2BufB/Z1dWlU6dOqaSkRB6PJ651AQCAMyzL0oULF1RRUSGvN/J5j7gbkaeeekrjxo3TiRMnhly2ra1NbW1tmjx5svLz8+P9KEnSqVOnNGHChITWBQAAzjp+/LiuuOKKiH8eVyNy4sQJPfbYY9q9e7dmzJgx5PKBQECFhYVasGCBtm/frhtuuEG//e1vNX78+IjrdDcv3boz+Y4fP67S0tJ4ygUAAA4JhUKaMGGCSkpKoi4XVyOyYsUKLVmyRNdcc01MywcCAbW2turWW2/Vr371K/3whz/UypUr9dvf/jbiOuvWrdMjjzwy4P3S0lIaEQAAMsxQt1XE3IjU1dXpww8/1Msvvxzzh0+ePFmnTp1SeXm5JGnZsmVasWJF1HVWrVqlBx54oOd1d0cFAADcJ+anZl599VU1NTWpoqJCI0aMUDAY1Jw5c/TKK69EXCcvL6+nCZGkkSNHKhQKRf2cgoKCnrMfnAUBAMDdYm5E1q9fr08++UQHDx7UwYMHVVJSoueff161tbUR19m4caNmzZrV8/rYsWPy+/1JFQwAANwj5kszY8aM0ZgxY3pee71elZWVafjw4QqFQioqKlJeXl6fdWbOnKkHH3xQ27Zt05QpU/T0009r0aJFthUPAIDpOjs71dHR4XQZtsvJyVFubm7SU2vE/fjuYL7xjW9ow4YNmjt3bp/3/+qv/kr/8R//oeXLl+vLL7/U3XffrX/6p3+y4yMBADDe559/rhMnTvQ8Aeo2xcXFKi8vT3iKDknyWIaPTigUks/nUzAY5H4RAEDG6Ozs1F/+8hcVFxdr7NixrpqU07Istbe36+zZs+rs7NSUKVMGTFoW6/e3LWdEAABAXx0dHbIsS2PHjlVRUZHT5diu+5aMo0ePqr29XYWFhQn9HkLvAABIITedCekv2tTtseKMCAAjdXZZ2tdwXmcutGpcSaGmV45Sjte9f6ED2YpGBIBx6g416ZGt9WoKtva8V+4r1Oo7qzR7anmUNQFkGi7NADBK3aEmLduyv08TIknNwVYt27JfdYeaHKoMyA779+/Xtddeq6KiIt1+++06c+ZMSj+PRgSAMTq7LD2ytV6DPcrX/d4jW+vV2WX0w36ArTq7LL3/v+f0/w6e1Pv/ey6lx39XV5e+973vac6cOfrLX/6ioqKilE+7waUZAMbY13B+wJmQy1mSmoKt2tdwXjOuHJ2+wgCHpPsy5R//+EedP39e//Zv/6bc3FytXr1aN910k7744gsNGzbM9s+TOCMCwCBnLkRuQhJZDshkTlym3L17t6ZPn67c3PB5imuvvVadnZ3av3+/7Z/VjUYEgDHGlcQ2D0GsywGZyqnLlM3NzQPiXEaOHKnTp0/b+jmXoxEBYIzplaNU7itUpId0PQqflp5eOSqdZQFpF89lSrv1n3DdsqyUzoVCIwLAGDlej1bfWSVJA5qR7ter76xiPhG4nlOXKcvLy3X27Nme152dnQoEAiorK7P1cy5HIwLAKLOnlmvTwmkq8/W9/FLmK9SmhdOYRwRZwanLlDU1Nfrggw906dIlSdKBAweUm5ur6667ztbPuRxPzQAwzuyp5bqtqoyZVZG1ui9TNgdbB71PxKNwc273ZcqbbrpJY8eO1erVq7Vs2TKtWbNGf/d3f6fi4mJbP+dynBEBYKQcr0czrhytu64drxlXjqYJQVZx6jKl1+vV7373O23btk1XXXWVWltb9eSTT9r6Gf1xRgQAAAN1X6bsP49IWYrjDqZNm6Y///nPKfndg6ERAQDAUNlwmZJGBAAAg3VfpnQr7hEBAACOoREBAACOoREBAACOoREBAACOoREBAACOoREBAACOoREBAMBkXZ1Swy7po/8b/t+uzpR/ZFNTk2655RYdPHgw5Z/FPCIAAJiq/g2p7p+l0Kne90orpNmPS1W1KfnIJUuW6LnnnkvJ7x4MZ0QAADBR/RvSaz/o24RIUqgp/H79Gyn52LVr16qhoSElv3swNCIAAJimqzN8JmTQ7N2v3qv7eUou04wZM0Z+v9/23xsJjQgAAKY5umfgmZA+LCl0MrxchqMRAQDANJ+ftnc5g9GIAABgmuFfs3c5g/HUDIABOrssV8eOA8abVB1+OibUpMHvE/GE/3xSdborsx2NCIA+6g416ZGt9WoKtva8V+4r1Oo7qzR7armDlQFZxJsTfkT3tR9I8qhvM/LVPwpmPxZeLsNxaQZAj7pDTVq2ZX+fJkSSmoOtWrZlv+oONTlUGZCFqmqleS9Jpf3+AVBaEX4/RfOIpBtnRABICl+OeWRrfcSHBT2SHtlar9uqyrhMA6RLVa309e+En475/HT4npBJ1Wk5E2JZg/1tYD8aEQCSpH0N5wecCbmcJakp2Kp9Dec148rR6SsMyHbeHKmyxukqUoZLMwAkSWcuRG5CElkOAGJBIwJAkjSupNDW5QAgFjQiACRJ0ytHqdxXqEh3f3gUfnpmeuWodJYFZLx03WvhBDu2jUYEgCQpx+vR6jurJGlAM9L9evWdVdyoCsQoJyd8Q2l7e7vDlaTOxYsXJUl5eXkJ/w5uVgXQY/bUcm1aOG3APCJlzCMCxC03N1fFxcU6e/as8vLy5PW659/+lmXp4sWLOnPmjEaMGNHTdCXCYxl+zigUCsnn8ykYDKq0tNTpcoCswMyqgD3a29vV0NCgrq4up0tJiREjRqisrEwez8C/H2L9/uaMCIABcrweHtEFbJCfn68pU6a48vJMXl5eUmdCutGIAACQQl6vV4WFPG0WiXsuWAEAgIzDGRHAJtxXAQDxoxEBbEBiLQAkhkszQJJIrAWAxNGIAEkYKrFWCifWdnYZ/ZQ8ADiGRgRIQjyJtQCAgWhEgCSQWAsAyaERAZJAYi0AJIdGBEgCibUAkBwaESAJJNYCQHJoRIAkdSfWlvn6Xn4p8xVq08JpzCMCAFEwoRlgg9lTy3VbVRkzqwJAnGhEAJuQWAsA8ePSDAAAcAyNCAAAcAyXZgC4FonIgPkSOiPy3nvvyePxaOfOnUMuu2PHDl199dUaNmyY5s+fr4sXLybykQAQl7pDTbrp8Xe0YPNerfg/B7Vg817d9Pg7hBAChom7Eeno6NB9990X07KBQEDz5s3TAw88oI8//liNjY1au3Zt3EUCQDxIRAYyR9yNyFNPPaVx48bJ5/MNuezrr7+uK664QkuXLpXf79dDDz2kLVu2JFQoAMSCRGQgs8TViJw4cUKPPfaYNm7cGNPyu3fvVnV1dc/rG2+8UceOHdPx48cjrtPW1qZQKNTnBwBiRSIykFniakRWrFihJUuW6Jprrolp+ebmZo0ZM6bn9ejR4TkWTp8+HXGddevWyefz9fxMmDAhnhIBZDkSkYHMEnMjUldXpw8//FAPP/xwXB9gWdaA//Z4It+1vmrVKgWDwZ6faGdPAKA/EpGBzBJzI/Lqq6+qqalJFRUVGjFihILBoObMmaNXXnkl4jrl5eU6e/Zsz+tz585JksrKyiKuU1BQoNLS0j4/ABArEpGBzBJzI7J+/Xp98sknOnjwoA4ePKiSkhI9//zzqq2tjbhOTU2N9uzZ0/N679698vv9Gj9+fHJVA0AEJCIDmSXmRmTMmDHy+/09P16vV2VlZRo+fLhCoZA6OjoGrHPXXXepqalJmzZtUmNjo9avX6+FCxfaugEA0B+JyEDmsGVm1W984xvasGGD5s6d2+d9n8+n1157TcuXL9eDDz6o2tparVq1yo6PBICoSEQGMoPHuvxuUgOFQiH5fD4Fg0HuFwEAIEPE+v1N6B0AAHAMjQgAAHAM6buAy7Rf6tLL7zfq6PmLmjSqWPfM8Cs/l39zADATjQjgIuu212vzrgZdHqOydvthLa6p1Ko7qpwrDAAioBEBXGLd9nr9+r2GAe93Wep5n2YEgGk4Xwu4QPulLm3eNbAJudzmXQ1qv9SVpooAIDY0IoALvPx+o4ZKte+ywssBgEloRAAXOHr+oq3LAUC60IgALjBpVLGtywFAutCIAC5wzwy/hpq53OsJLwcAJqERAVwgP9erxTWVUZdZXFPJfCIAjMPju4BLdD+a238eEa9HzCMCwFiE3gEuw8yqAEwQ6/c3Z0QAl8nP9eremslOlwEAMeGfSQAAwDE0IgAAwDFcmgG+8mV7px7dXq/GcxflH12sf7mjSkX5OU6XlbU6uyztazivMxdaNa6kUNMrRylnqGeUAWQcGhFA0uKXPtB/1Z/peb3rL9LLe4/ptqpx2vyDGxysLDvVHWrSI1vr1RRs7Xmv3Feo1XdWafbUcgcrA2A3Ls0g6/VvQi73X/VntPilD9JcUXarO9SkZVv292lCJKk52KplW/ar7lCTQ5UBSAUaEWS1L9s7IzYh3f6r/oy+bO9MU0XZrbPL0iNb6zXYnALd7z2ytV6dQyX8AcgYNCLIao9ur7d1OSRnX8P5AWdCLmdJagq2al/D+fQVBSClaESQ1RrPxZZGG+tySM6ZC5GbkESWA2A+GhFkNf/o2NJoY10OyRlXUmjrcgDMRyOCrPYvMeavxLockjO9cpTKfYWK9JCuR+GnZ6ZXjkpnWQBSiEYEWa0oP0e3VY2LusxtVeOYTyRNcrwerb4z3PT1b0a6X6++s4r5RAAXoRFB1tv8gxsiNiPMI5J+s6eWa9PCaSrz9b38UuYr1KaF05hHBHAZ0neBrzCzqlmYWRXIbLF+f9OIAAAA28X6/c2lGQAA4BgaEQAA4BhC74CvmHBPgh01mLAdABArGhFAZqS92lGDCdsBAPHg0gyynglpr3bUYMJ2AEC8aESQ1UxIe7WjBhO2AwASQSOCrGZC2qsdNZiwHQCQCBoRZDUT0l7tqMGE7QCARNCIIKuZkPZqRw0mbAcAJIJGBFnNhLRXO2owYTsAIBE0IshqJqS92lGDCdsBAImgEUHWMyHt1Y4aTNgOAIgXoXfAV0yYkZSZVQG4Razf38ysCnwlx+vRjCtHZ3wNJmwHAMSKSzMAAMAxNCIAAMAxXJrJYG65F4D7IgAge9GIZCi3pKySOAsA2Y1LMxnILSmrJM4CAGhEMoxbUlZJnAUASDQiGcctKaskzgIAJBqRjOOWlFUSZwEAEo1IxnFLyiqJswAAiUYk47glZZXEWQCARCOScdySskriLABAohHJSG5JWSVxFgBA+m4Gc8tsosysCgDuQ/puFnBLyiqJswCQvbg0AwAAHEMjAgAAHMOlGbhC+6Uuvfx+o46ev6hJo4p1zwy/8nPj67OT/R1uuk/FTdsCwGxx36x6+PBh/ehHP9JHH32k6667Ti+88IKmTJkSdZ2ysjKdPn265/X111+vP/3pTzF9HjerYijrttdr864GXR4p4/VIi2sqteqOqrT8DjclALtpWwA4J9bv77gvzXz/+99XbW2tPv30U33961/X0qVLh1wnEAjo8OHDamlpUUtLi3bu3BnvxwKDWre9Xr9+r28DIUldlvTr9xq0bnt9yn+HmxKA3bQtADJDXI1IS0uLhg8frgcffFAVFRX67ne/q/r66H9Jt7W1qa2tTZMnT9aIESM0YsQIDR8+PKmiASl8KWXzroaoy2ze1aD2S10p+x1uSgB207YAyBxxNSIjR47Url27lJ+fr/b2dr322mu67rrroq4TCARUWFioBQsWqKioSDfffLNOnjwZcfm2tjaFQqE+P8BgXn6/ccBZjP66rPByqfodbkoAdtO2AMgcCT81U1xcrLfeekvPPvts1OUCgYBaW1t16623qr6+Xl6vVytXroy4/Lp16+Tz+Xp+JkyYkGiJcLmj5y8mvVyyv8NNCcBu2hYAmSPhRmTfvn2aNm2a7r///qjLTZ48WadOndJ9992nyspKLVu2TO+++27E5VetWqVgMNjzc/z48URLhMtNGlWc9HLJ/g43JQC7aVsAZI64GpGzZ8/qwIEDkqRp06bp0Ucf1ZtvvqlgMBhxnby8PJWX995pP3LkyKiXWwoKClRaWtrnBxjMPTP8GuqJUq8nvFyqfoebEoDdtC0AMkdcjciBAwf0ne98p+e1xxP+K8vrjfxrNm7cqFmzZvW8PnbsmPx+f5xlAgPl53q1uKYy6jKLayqjzgWS7O9wUwKwm7YFQOaIqxGZPn26WltbtXHjRp04cUJPPvmkampqVFJSolAopI6OjgHrzJw5U3v27NG2bdv0ySef6Omnn9aiRYvsqh9ZbtUdVVpyc+WAsxpej7Tk5tjmAEn2d7gpAdhN2wIgM8Q9odnOnTu1YsUKHTlyRNXV1Xruuec0adIk+f1+bdiwQXPnzh2wzm9+8xutXr1aX375pe6++2498cQTysvLi+nzmNAMsWBmVXu5aVsAOCPW7++4G5F0oxEBACDzpGxmVQAAALvQiAAAAMeQvpvBTLmOb8f9GSbUkOx4sj8wqK5O6ege6fPT0vCvSZOqJW+O01UBxuAekQxlSkKqHcm3JtSQ7HiyPzCo+jekun+WQqd63yutkGY/LlXVOlcXkAbcrOpi3Qmp/Xdc97+90/WYZXdqbSSxPj7rdA3Jjif7A4Oqf0N67QdSpCNj3ks0I3A1blZ1KVMSUu1IvjWhhmTHk/2BQXV1hs+ERDsy6n4eXg7IcjQiGcaUhFQ7km9NqCHZ8WR/YFBH9/S9HDOAJYVOhpcDshyNSIYxJSHVjuRbE2pIdjzZHxjU56ftXQ5wMRqRDGNKQqodybcm1JDseLI/MKjhX7N3OcDFaEQyjCkJqXYk35pQQ7Ljyf7AoCZVh5+OiXZklI4PLwdkORqRDGNKQqodybcm1JDseLI/MChvTvgRXUkRj4zZjzGfCCAakYxkSkKqHcm3JtSQ7HiyPzCoqtrwI7ql/fZ/aQWP7gKXYR6RDMZMnvbWwMyqSAlmVkWWYkIzAADgGCY0AwAAxqMRAQAAjiF9F65gx/0ZptzjARiJe12QIjQiyHh2JN+akp4LGIkUYaQQl2aQ0bqTb/vnvTQHW7Vsy37VHWpKy+8AXKs7Rbh/dk6oKfx+/RvO1AXXoBFBxrIj+daU9FzASKQIIw1oRJCx7Ei+NSU9FzASKcJIAxoRZCw7km9NSc8FjESKMNKARgQZy47kW1PScwEjkSKMNKARQcayI/nWlPRcwEikCCMNaESQsexIvjUlPRcwEinCSAMaEWQ0O5JvTUnPBYxEijBSjNA7uAIzqwIpxsyqiFOs39/MrApXyPF6NOPK0Y7/DsC1vDlSZY3TVcCFuDQDAAAcQyMCAAAcw6WZBJlwP4EdNbRf6tLL7zfq6PmLmjSqWPfM8Cs/N/P6UxP2B1yI+yLsxXiaxZD9QSOSABOSWu2oYd32em3e1aDLY1TWbj+sxTWVWnVHld0lp4wJ+wMuROKsvRhPsxi0P3hqJk7dSa39B637397peNzTjhrWba/Xr99riPjnS27OjGbEhP0BF+pOnI10ZPHYanwYT7OkaX/E+v2deefgHWRCUqsdNbRf6tLmXZGbEEnavKtB7Ze6Ei80DUzYH3AhEmftxXiaxcD9QSMSBxOSWu2o4eX3GzXUd3OXFV7OZCbsD7gQibP2YjzNYuD+oBGJgwlJrXbUcPT8xZh+R6zLOcWE/QEXInHWXoynWQzcHzQicTAhqdWOGiaNKo7pd8S6nFNM2B9wIRJn7cV4msXA/UEjEgcTklrtqOGeGX4N9WSr1xNezmQm7A+4EImz9mI8zWLg/qARiYMJSa121JCf69Ximsqon7O4ptL4+URM2B9wIRJn7cV4msXA/WH2N42BTEhqtaOGVXdUacnNlQPOjHg9mfPormTG/oALkThrL8bTLIbtD+YRSZAJM3kys2ovE/YHXMiQmSddg/E0S4r3R6zf3zQiAADAdkxoBgAAjEcjAgAAHEPoXQYz5b6IZOtwy30qALLEpXbpg81SS6M00i/dsFjKzU9vDS6634Z7RDKUKYmzydYxWAKw16OMSwAGkCXeflh6/1nJuiyLy+OVZvxUuv2X6anBoOTcaLhHxMW6E2f756w0B1u1bMt+1R1qyog6uhOA++fedFnSr99r0Lrt9XaXDACJe/thac8zfZsQKfx6zzPhP0+17uTc/nkxoabw+/VvpL4Gm9GIZBhTEmeTrcMtCcAAssSl9vCZkGje3xheLlUMTM61A41IhjElcTbZOtySAAwgS3yweeCZkP6szvByqWJgcq4daEQyjCmJs8nW4ZYEYABZoqXR3uUSYWByrh1oRDKMKYmzydbhlgRgAFlipN/e5RJhYHKuHWhEMowpibPJ1uGWBGAAWeKGxeGnY6Lx5ISXSxUDk3PtQCOSYUxJnE22DrckAAPIErn54Ud0o5mxPLXziRiYnGsH/pbPQKYkziZbh1sSgAFkidt/KVXfP/DMiCcn/H465hExLDnXDkxolsGYWRUAHMDMqjEhfRcAADiGmVUBAIDxaEQAAIBjsjJ91457K0y5P8MEyd7jwf6wmQnXju24hm7CdphQg0l1uIEdY8n+sFXc94gcPnxYP/rRj/TRRx/puuuu0wsvvKApU6ZEXWfHjh36x3/8R508eVJ33nmnXnjhBRUXxzZRld33iNiRWmtK8q0Jkk3PZX/YzIRUTjvSSU3YDhNqMKkON7BjLNkfMUvZPSLf//73VVtbq08//VRf//rXtXTp0qjLBwIBzZs3Tw888IA+/vhjNTY2au3atfF+rC3sSK01JfnWBMmm57I/bGZCKqcd6aQmbIcJNZhUhxvYMZbsj5SIqxFpaWnR8OHD9eCDD6qiokLf/e53VV8f/cvm9ddf1xVXXKGlS5fK7/froYce0pYtW5IqOhF2pNaaknxrgmTTc9kfNjMhldOOdFITtsOEGkyqww3sGEv2R8rE1YiMHDlSu3btUn5+vtrb2/Xaa6/puuuui7rO7t27VV3dO93sjTfeqGPHjun48eODLt/W1qZQKNTnxw52pNaaknxrgmTTc9kfNjMhldOOdFITtsOEGkyqww3sGEv2R8ok/NRMcXGx3nrrLT37bPR/ATU3N2vMmDE9r0ePHi1JOn168HTAdevWyefz9fxMmDAh0RL7sCO11pTkWxMkm57L/rCZCamcdqSTmrAdJtRgUh1uYMdYsj9SJuFGZN++fZo2bZruv//+IZe9/H7Y7v/2eAZ/omHVqlUKBoM9P5HOnMTLjtRaU5JvTZBsei77w2YmpHLakU5qwnaYUINJdbiBHWPJ/kiZuBqRs2fP6sCBA5KkadOm6dFHH9Wbb76pYDAYcZ3y8nKdPXu25/W5c+ckSWVlZYMuX1BQoNLS0j4/drAjtdaU5FsTJJuey/6wmQmpnHakk5qwHSbUYFIdbmDHWLI/UiauRuTAgQP6zne+0/O6+6yG1xv519TU1GjPnt5rZnv37pXf79f48ePjrTUpdqTWmpJ8a4Jk03PZHzYzIZXTjnRSE7bDhBpMqsMN7BhL9kfKxNWITJ8+Xa2trdq4caNOnDihJ598UjU1NSopKVEoFFJHR8eAde666y41NTVp06ZNamxs1Pr167Vw4ULbNiAedqTWmpJ8a4Jk03PZHzYzIZXTjnRSE7bDhBpMqsMN7BhL9kdKxD2h2c6dO7VixQodOXJE1dXVeu655zRp0iT5/X5t2LBBc+fOHbDO22+/reXLl+vkyZOqra11dEIziZk87cbMqoYxYdZHZlZ1Zx1uwMyqaUP6LgAAcAzpuwAAwHg0IgAAwDFZmb5rB+5JAKLgOnwvE8bClLE0oQ4TakAfNCIJIO0ViIKE014mjIUpY2lCHSbUgAG4WTVO3Wmv/Qet+1xI1j0yClyuO5000v9DYnnE0Y7fYQITxsKUsTShDhNqyDLcrJoCpL0CUZBw2suEsTBlLE2ow4QaEBGNSBxIewWiIOG0lwljYcpYmlCHCTUgIhqROJD2CkRBwmkvE8bClLE0oQ4TakBENCJxIO0ViIKE014mjIUpY2lCHSbUgIhoROJA2isQBQmnvUwYC1PG0oQ6TKgBEdGIxIG0VyAKEk57mTAWpoylCXWYUAMiohGJE2mvQBQknPYyYSxMGUsT6jChBgyKeUQSxMyqQBQmzCZqChPGwpSxNKEOE2rIEqTvAgAAxzChGQAAMB6NCAAAcAyhdwDMdKld+mCz1NIojfRLNyyWcvOdrsoZjEUvt9zj4ZbtsAH3iAAwz9sPS+8/K1ldve95vNKMn0q3/9K5upzAWPRyS3quW7ZjCNwjAiAzvf2wtOeZvl+8Uvj1nmfCf54tGIte3em5/TNjQk3h9+vfcKaueLllO2xEIwLAHJfaw//6j+b9jeHl3I6x6OWW9Fy3bIfNaEQAmOODzQP/9d+f1Rlezu0Yi15uSc91y3bYjEYEgDlaGu1dLpMxFr3ckp7rlu2wGY0IAHOM9Nu7XCZjLHq5JT3XLdthMxoRAOa4YXH4iZBoPDnh5dyOsejllvRct2yHzWhEAJgjNz/8WGo0M5ZnxxwajEUvt6TnumU7bEYjAsAst/9Sqr5/4NkAT074/WyaO4Ox6OWW9Fy3bIeNmNAMgJmYTbQXY9HLLTOSumU7oiB9FwAAOIaZVQEAgPFoRAAAgGNI3wXskgXXfGNmyliYcG+FKWMBGIpGBLBDlqRpxsSUsRgstfbtX6Q3tdaUsQAMxqUZIFmkafYyZSxMSK01ZSwAw9GIAMkgTbOXKWNhQmqtKWMBZAAaESAZpGn2MmUsTEitNWUsgAxAIwIkgzTNXqaMhQmptaaMBZABaESAZJCm2cuUsTAhtdaUsQAyAI0IkAzSNHuZMhYmpNaaMhZABqARAZJBmmYvU8bChNRaU8YCyAA0IkCySNPsZcpYmJBaa8pYAIYj9A6wCzNo9jJlLJhZFXAM6bsAAMAxpO8CAADj0YgAAADHEHoHYCAT7muwowYTtgNAVDQiAPoyITHWjhpM2A4AQ+LSDIBeJiTG2lGDCdsBICY0IgDCTEiMtaMGE7YDQMxoRACEmZAYa0cNJmwHgJjRiAAIMyEx1o4aTNgOADGjEQEQZkJirB01mLAdAGJGIwIgzITEWDtqMGE7AMSMRgRAmAmJsXbUYMJ2AIgZjQiAXiYkxtpRgwnbASAmhN4BGMiEGUmZWRXIaLF+fzOzKoCBvDlSZU3m12DCdgCIikszAADAMTQiAADAMVyaAbpxP0GvZMeCsXQf9ilSJO5G5MiRI/rhD3+o/fv36/rrr9eLL76oSZMmRV2nrKxMp0/3zmJ4/fXX609/+lP81QKpQlJrr2THgrF0H/YpUijuSzM/+clPNHHiRB06dEijR4/W8uXLh1wnEAjo8OHDamlpUUtLi3bu3JlIrUBqkNTaK9mxYCzdh32KFIvr8d329nYVFhbq0KFDqqqq0vbt27VgwQIFg8GI67S1tamwsFBtbW3Kz8+Pu0Ae30VKdXVKG6ZGCUnzhP/l97OP3H8aOtmxYCzdh32KJMT6/R3XGZGOjg498cQTqqyslCSdO3dORUVFUdcJBAIqLCzUggULVFRUpJtvvlknT56MuHxbW5tCoVCfHyBlSGrtlexYMJbuwz5FGsTViAwbNkwrV65UUVGROjo69Mwzz+iee+6Juk4gEFBra6tuvfVW1dfXy+v1auXKlRGXX7dunXw+X8/PhAkT4ikRiA9Jrb2SHQvG0n3Yp0iDhB7fvXTpku6++255vV6tWbMm6rKTJ0/WqVOndN9996myslLLli3Tu+++G3H5VatWKRgM9vwcP348kRKB2JDU2ivZsWAs3Yd9ijSI+6mZrq4uzZ8/X0eOHNEf/vCHIS/N5OXlqby8N+9h5MiRUS+3FBQUqKCgIN6ygMR0J7WGmiQNdrvUV9fAsyGpNdmxYCzdh32KNIj7jMiaNWv02Wef6Z133tGoUaOGXH7jxo2aNWtWz+tjx47J7/fH+7FAapDU2ivZsWAs3Yd9ijSIqxFpbm7WU089pU2bNkkK3/8RCATU1dWlUCikjo6OAevMnDlTe/bs0bZt2/TJJ5/o6aef1qJFi2wpHrAFSa29kh0LxtJ92KdIsbge333xxRcHbSIaGho0c+ZMbdiwQXPnzh3w57/5zW+0evVqffnll7r77rv1xBNPKC8vL6bP5PFdpA0zR/ZiZlX0xz5FnGL9/o6rEXECjQgAAJknJfOIAAAA2IlGBAAAOIb0XTiPa8/mudQufbBZammURvqlGxZLufFHNADAUGhE4CxSPc3z9sPS+89KVtdl7/1CmvFT6fZfOlcXAFfi0gycQ6qned5+WNrzTN8mRAq/3vNM+M8BwEY0InBGV2f4TMigszV+9V7dz8PLIT0utYfPhETz/sbwcgBgExoROINUT/N8sHngmZD+rM7wcgBgExoROINUT/O0NNq7HADEgEYEziDV0zwj/fYuBwAxoBGBM7pTPQcEaXXzSKXjSfVMpxsWS54h/krw5ISXAwCb0IjAGaR6mic3P/yIbjQzljOfCABb0YjAOaR6muf2X0rV9w88M+LJCb/PPCIAbEboHZzHzKrmYWZVAEmK9fubmVXhPG+OVFnjdBW4XG5++DIMAKQYl2YAAIBjaEQAAIBjuDTjoM4uS/sazuvMhVaNKynU9MpRyvFGepwVUXGfib0YT/THMYEUoRFxSN2hJj2ytV5Nwdae98p9hVp9Z5VmTy2PsiYGIMHXXown+uOYQApxacYBdYeatGzL/j5NiCQ1B1u1bMt+1R1qcqiyDESCr70YT/THMYEUoxFJs84uS49srY+WOatHttars8vop6rNQIKvvRhP9McxgTSgEUmzfQ3nB5wJuZwlqSnYqn0N59NXVKYiwddejCf645hAGtCIpNmZC5GbkESWy2ok+NqL8UR/HBNIAxqRNBtXUmjrclmNBF97MZ7oj2MCaUAjkmbTK0ep3FcYLXNW5b7wo7wYAgm+9mI80R/HBNKARiTNcrwerb6zSlLEzFmtvrOK+URiQYKvvRhP9McxgTSgEXHA7Knl2rRwmsp8fS+/lPkKtWnhNOYRiQcJvvZiPNEfxwRSjPRdBzGzqo2Y9dFejCf645hAnGL9/qYRAQAAtov1+5tLMwAAwDE0IgAAwDGE3gFAKplwb4UJNQAR0IgAQKqYkFprQg1AFFyaAYBUMCG11oQagCHQiACA3UxIrTWhBiAGNCIAYDcTUmtNqAGIAY0IANjNhNRaE2oAYkAjAgB2MyG11oQagBjQiACA3UxIrTWhBiAGNCIAYDcTUmtNqAGIAY0IAKSCCam1JtQADIHQOwBIJRNmNTWhBmSdWL+/mVkVAFLJmyNV1lADEAGXZgAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNoRAAAgGNynS4AsEVXp3R0j/T5aWn416RJ1ZI3x+mqAABDiPuMyJEjR3TLLbeopKREM2fO1NGjR4dcZ8eOHbr66qs1bNgwzZ8/XxcvXkyoWGBQ9W9IG6ZKL86Rfn9v+H83TA2/DwAwWtyNyE9+8hNNnDhRhw4d0ujRo7V8+fKoywcCAc2bN08PPPCAPv74YzU2Nmrt2rUJFwz0Uf+G9NoPpNCpvu+HmsLv04wAgNHiakTa29v1zjvvaNWqVZo0aZLuvfde7dq1K+o6r7/+uq644gotXbpUfr9fDz30kLZs2ZJU0YCk8OWYun+WZA3yh1+9V/fz8HIAACPF1Yh0dHToiSeeUGVlpSTp3LlzKioqirrO7t27VV1d3fP6xhtv1LFjx3T8+PFBl29ra1MoFOrzAwzq6J6BZ0L6sKTQyfByAAAjxdWIDBs2TCtXrlRRUZE6Ojr0zDPP6J577om6TnNzs8aMGdPzevTo0ZKk06dPD7r8unXr5PP5en4mTJgQT4nIJp8PfgwlvBwAIO0Senz30qVLuvvuu+X1erVmzZohl7csa8B/ezyeQZddtWqVgsFgz0+kMyeAhn/N3uUAAGkX9+O7XV1dmj9/vo4cOaI//OEPQ16aKS8v19mzZ3tenzt3TpJUVlY26PIFBQUqKCiItyxko0nVUmlF+MbUQe8T8YT/fFL1IH8GADBB3GdE1qxZo88++0zvvPOORo0aNeTyNTU12rOn9xr93r175ff7NX78+Hg/GujLmyPNfvyrF/3PsH31evZjzCcCAAaLqxFpbm7WU089pU2bNkkKP5obCATU1dWlUCikjo6OAevcddddampq0qZNm9TY2Kj169dr4cKF9lQPVNVK816SSsv7vl9aEX6/qtaZugAAMfFYl9/AMYQXX3xRixYtGvB+Q0ODZs6cqQ0bNmju3LkD/vztt9/W8uXLdfLkSdXW1uqFF15QcXFxTJ8ZCoXk8/kUDAZVWloaa6nINsysCgBGifX7O65GxAk0IgAAZJ5Yv78JvQMAAI6hEQEAAI6hEQEAAI6hEQEAAI6hEQEAAI6hEQEAAI6hEQEAAI6hEQEAAI6hEQEAAI6JO3033bonfg2FQg5XAgAAYtX9vT3UBO7GNyIXLlyQJE2YMMHhSgAAQLwuXLggn88X8c+Nz5rp6urSqVOnVFJSIo+nf9R75guFQpowYYKOHz9Olk6SGEt7MZ72YSztxXjaJ5VjaVmWLly4oIqKCnm9ke8EMf6MiNfr1RVXXOF0GSlXWlrK/6Fswljai/G0D2NpL8bTPqkay2hnQrpxsyoAAHAMjQgAAHAMjYjDCgoKtHr1ahUUFDhdSsZjLO3FeNqHsbQX42kfE8bS+JtVAQCAe3FGBAAAOIZGBAAAOIZGBAAAOIZGBAAAOIZGJA3ee+89eTwe7dy5c8hly8rK5PF4en6++c1vpr7ADPGtb32rz9iMGTNmyHV27Nihq6++WsOGDdP8+fN18eLFNFSaGRIZT47PwXV0dGjJkiUqKSlRVVWV9u3bN+Q6HJuRJTKeHJsDNTY29hmT7p+hpPvYpBFJsY6ODt13330xLx8IBHT48GG1tLSopaUlpuYlWwQCAe3YsaNnbI4cOTLk8vPmzdMDDzygjz/+WI2NjVq7dm2aqjVfvOPZvQ7H50C/+tWv1NjYqAMHDmj+/PlauHBh1OU5NqOLdzwljs3BTJw4sWc8Wlpa9NJLL6msrCzqOo4cmxZS6vHHH7f+5m/+xvL5fNa7774bddnW1lZLktXW1pae4jJMWVmZ9emnn8a8/AsvvGBVVVX1vP79739vTZw4MRWlZaR4x5PjM7Irr7zSOnjwoGVZlnXhwgXrd7/7ndXZ2RlxeY7N6OIdT47N2Pz4xz+2Fi1aFHUZJ45Nzoik0IkTJ/TYY49p48aNMS0fCARUWFioBQsWqKioSDfffLNOnjyZ4iozRyAQ0L/+67+qqKhI1157rT7++OOoy+/evVvV1dU9r2+88UYdO3ZMx48fT3WpGSHe8eT4HFxzc7OOHDmiP/7xj/L5fLrlllv013/911FDvjg2I0tkPDk2Y/PWW2/pjjvuiLqME8cmjUgKrVixQkuWLNE111wT0/KBQECtra269dZbVV9fL6/Xq5UrV6a4yszQ3t6u1tZWXXXVVaqvr1dVVZV+/OMfR12nubm5z30Po0ePliSdPn06pbVmgkTGk+NzcE1NTfJ6vfrv//5v/fnPf9Y111yjpUuXRl2HYzOyRMaTY3NoBw8e1OnTp3XbbbdFXc6JY9P49N1MVVdXpw8//FAvv/xyzOtMnjxZp06dUnl5uSRp2bJlWrFiRapKzCh5eXk6ceKExo8fLync5H3rW9/Sl19+qaKioojrWZdNHNz937HcrOV2iYwnx+fgvvjiC3V2dmr16tXy+/366U9/qurqarW3tys/Pz/iehybg0tkPDk2h7Z9+3bNmDFDI0aMGHLZdB+bnBFJkVdffVVNTU2qqKjQiBEjFAwGNWfOHL3yyisR18nLy+v5P5IkjRw5UqFQKB3lGs/j8fR8aUrhsZGkCxcuRFynvLxcZ8+e7Xl97tw5SRryZq1skMh4cnwOrjvmfNSoUZLC/4K0LEvnz5+PuA7HZmSJjCfH5tC2b9+uv/3bvx1yOSeOTRqRFFm/fr0++eQTHTx4UAcPHlRJSYmef/551dbWRlxn48aNmjVrVs/rY8eOye/3p6Fa823btk1XXnllz+tjx46puLhYY8eOjbhOTU2N9uzZ0/N679698vv9fb6As1Ui48nxObirrrpKeXl5+vTTTyWFT2Hn5OREfRyaYzOyRMaTYzO6lpYW7d27d8j7QySHjs2U3gqLHpc/NRMMBq329vYByxw6dMgqKCiwtm7dav3P//yPNXXqVOvxxx9Pc6VmOnPmjDV8+HBr8+bN1pEjR6xZs2ZZy5Ytsywr8ngGAgHL5/NZ//7v/241NDRY06dPt37xi1+ku3QjJTKeHJ+Rfe9737Nuu+0267PPPrP+/u//3pozZ45lWRybiYp3PDk2o3vllVesioqKPu+ZdGzSiKTJ5Y3IpEmTrNdff33Q5f7zP//TmjhxojV27FjrZz/72aAHSrZ68803rauvvtoaMWKEdc8991gXLlywLCv6eO7YscO66qqrrKKiIusf/uEfrC+++CKNFZstkfHk+Bzc6dOnrW9/+9tWUVGRVVNTYzU2NlqWxbGZqETGk2MzsoULF1r33ntvn/dMOjY9lnXZXSkAAABpxD0iAADAMTQiAADAMTQiAADAMTQiAADAMTQiAADAMTQiAADAMTQiAADAMTQiAADAMTQiAADAMTQiAADAMf8fvVfCTXn5zI4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.datasets import load_iris\n",
    "from sklearn.model_selection  import train_test_split\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# data\n",
    "def create_data():\n",
    "    iris = load_iris()\n",
    "    df = pd.DataFrame(iris.data, columns=iris.feature_names)\n",
    "    df['label'] = iris.target\n",
    "    df.columns = ['sepal length', 'sepal width', 'petal length', 'petal width', 'label']\n",
    "    data = np.array(df.iloc[:100, [0, 1, -1]])\n",
    "    for i in range(len(data)):\n",
    "        if data[i,-1] == 0:\n",
    "            data[i,-1] = -1\n",
    "    # print(data)\n",
    "    return data[:,:2], data[:,-1]\n",
    "\n",
    "X, y = create_data()\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)\n",
    "\n",
    "plt.scatter(X[:50,0],X[:50,1], label='0')\n",
    "plt.scatter(X[50:,0],X[50:,1], label='1')\n",
    "plt.legend()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "436e4deb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The score used AdaBoost to predict is 60.00%\n"
     ]
    }
   ],
   "source": [
    "clf = AdaBoost(n_estimators=10, learning_rate=0.2)\n",
    "clf.fit(X_train, y_train)\n",
    "print(f'The score used AdaBoost to predict is {clf.score(X_test, y_test)*100:.2f}%')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2b54b9bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average score:65.303%\n"
     ]
    }
   ],
   "source": [
    "# 100次结果\n",
    "result = []\n",
    "for i in range(1, 101):\n",
    "    X, y = create_data()\n",
    "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33)\n",
    "    clf = AdaBoost(n_estimators=100, learning_rate=0.2)\n",
    "    clf.fit(X_train, y_train)\n",
    "    r = clf.score(X_test, y_test)\n",
    "    # print('{}/100 score：{}'.format(i, r))\n",
    "    result.append(r)\n",
    "\n",
    "print('average score:{:.3f}%'.format(sum(result)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a5f18e",
   "metadata": {},
   "source": [
    "# scikit-learn实例\n",
    "## sklearn.ensemble.AdaBoostClassifier\n",
    "\n",
    "- algorithm：这个参数只有AdaBoostClassifier有。主要原因是scikit-learn实现了两种Adaboost分类算法，SAMME和SAMME.R。两者的主要区别是弱学习器权重的度量，SAMME使用了和我们的原理篇里二元分类Adaboost算法的扩展，即用对样本集分类效果作为弱学习器权重，而SAMME.R使用了对样本集分类的预测概率大小来作为弱学习器权重。由于SAMME.R使用了概率度量的连续值，迭代一般比SAMME快，因此AdaBoostClassifier的默认算法algorithm的值也是SAMME.R。我们一般使用默认的SAMME.R就够了，但是要注意的是使用了SAMME.R， 则弱分类学习器参数base_estimator必须限制使用支持概率预测的分类器。SAMME算法则没有这个限制。\n",
    "\n",
    "\n",
    "- n_estimators： AdaBoostClassifier和AdaBoostRegressor都有，就是我们的弱学习器的最大迭代次数，或者说最大的弱学习器的个数。一般来说n_estimators太小，容易欠拟合，n_estimators太大，又容易过拟合，一般选择一个适中的数值。默认是50。在实际调参的过程中，我们常常将n_estimators和下面介绍的参数learning_rate一起考虑。\n",
    "\n",
    "\n",
    "-  learning_rate:  AdaBoostClassifier和AdaBoostRegressor都有，即每个弱学习器的权重缩减系数ν\n",
    "\n",
    "\n",
    "- base_estimator：AdaBoostClassifier和AdaBoostRegressor都有，即我们的弱分类学习器或者弱回归学习器。理论上可以选择任何一个分类或者回归学习器，不过需要支持样本权重。我们常用的一般是CART决策树或者神经网络MLP。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97136768",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "clf = AdaBoostClassifier(n_estimators=100, learning_rate=0.5)\n",
    "clf.fit(X_train, y_train)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
